Design and Implementation of an Efficient Distributed Web Crawler with Scalable Architecture

Distributed Web crawlers have recently received more and more attention from researchers. Centralized solutions are known to have problems like link congestion, being a single point of failure ,while the fully distributed crawlers become an interesting architectural paradigm for its scalability, increased autonomy of nodes. This paper provides a distributed crawler system which consists of multiple controllers and takes the advantages of both two architecture.The design involves a fully distributed architecture,a strategy to assign tasks and a method to assure system scalability. Finally, an experimental study is used to verify the advantages of our crawler, and the results are comparatively satisfying.